<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" face="Menlo">Hi,</font><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo">I found the terrible situation on our lustre system.</font></div><div><font class="Apple-style-span" face="Menlo">A OST (raid 6: 8+2, spare 1) had 2 disk failures almost at the same time. While recovering it, another disk failed. so recovering procedure seems to be halt, and the spare disk which were in resync fell into "spare" status again. (I guess that resync procedure almost finished more than 95%) </font></div><div><font class="Apple-style-span" face="Menlo">Right now we have just 7 disks for this md. Is there any possibility to recover from this situation?</font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo">The following is detailed log.</font></div><div><font class="Apple-style-span" face="Menlo">#1 the original configuration before any failure</font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"> Number Major Minor RaidDevice State</font></div><div><font class="Apple-style-span" face="Menlo"> 0 8 176 0 active sync /dev/sdl</font></div><div><font class="Apple-style-span" face="Menlo"> 1 8 192 1 active sync /dev/sdm</font></div><div><font class="Apple-style-span" face="Menlo"> 2 8 208 2 active sync /dev/sdn</font></div><div><font class="Apple-style-span" face="Menlo"> 3 8 224 3 active sync /dev/sdo</font></div><div><font class="Apple-style-span" face="Menlo"> 4 8 240 4 active sync /dev/sdp</font></div><div><font class="Apple-style-span" face="Menlo"> 5 65 0 5 active sync /dev/sdq</font></div><div><font class="Apple-style-span" face="Menlo"> 6 65 16 6 active sync /dev/sdr</font></div><div><font class="Apple-style-span" face="Menlo"> 7 65 32 7 active sync /dev/sds</font></div><div><font class="Apple-style-span" face="Menlo"> 8 65 48 8 active sync /dev/sdt</font></div><div><font class="Apple-style-span" face="Menlo"> 9 65 96 9 active sync /dev/sdw</font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"> 10 65 64 - spare /dev/sdu</font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo">#2 a disk(sdl) failed, and resync started after adding spare disk(sdu)</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel: sd 1:0:10:0: SCSI error: return code = 0x08000002</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel: sdl: Current: sense key: Medium Error</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel: Add. Sense: Unrecovered read error</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel: Info fld=0x74241ace</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:53:33 oss07 kernel: end_request: I/O error, dev sdl, sector 1948523214</font></div></div><div><font class="Apple-style-span" face="Menlo">... ...</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:15 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: --- rd:10 wd:9 fd:1</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 1, o:1, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 4, o:1, dev:sdp</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: --- rd:10 wd:9 fd:1</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 0, o:1, dev:sdu</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 1, o:1, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 4, o:1, dev:sdp</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:16 oss07 kernel: md: syncing RAID array md12</font></div></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo">#3 another disk(sdp) failed</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:42 oss07 kernel: end_request: I/O error, dev sdp, sector 1949298688</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:42 oss07 kernel: mptbase: ioc1: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ FaCommands After Error}, SubCode(0x0000)</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:42 oss07 last message repeated 3 times</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:42 oss07 kernel: raid5:md12: read error not correctable (sector 1949298688 on sdp).</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:42 oss07 kernel: raid5: Disk failure on sdp, disabling device. Operation continuing on</font></div></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:43 oss07 kernel: end_request: I/O error, dev sdp, sector 1948532499</font></div><div><font class="Apple-style-span" face="Menlo">... ...</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:44 oss07 kernel: raid5:md12: read error not correctable (sector 1948532728 on sdp).</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:44 oss07 kernel: md: md12: sync done.</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: --- rd:10 wd:8 fd:2</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 0, o:1, dev:sdu</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 1, o:1, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 4, o:0, dev:sdp</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:53 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">... ...</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: --- rd:10 wd:8 fd:2</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 1, o:1, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: --- rd:10 wd:8 fd:2</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 0, o:1, dev:sdu</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 1, o:1, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:54 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:55 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:55 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">May 7 04:54:55 oss07 kernel: md: syncing RAID array md12</font></div></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"># the 3rd disk(sdm) failed while resyncing</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 09:41:53 oss07 kernel: mptbase: ioc1: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:41:57 oss07 kernel: mptbase: ioc1: LogInfo(0x31110e00): Originator={PL}, Code={Reset}, SubCode(0x0e00)</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:41:59 oss07 last message repeated 24 times</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:04 oss07 kernel: mptbase: ioc1: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 last message repeated 43 times</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: sd 1:0:11:0: SCSI error: return code = 0x000b0000</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: end_request: I/O error, dev sdm, sector 1948444160</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: mptbase: ioc1: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 last message repeated 3 times</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: raid5:md12: read error not correctable (sector 1948444160 on sdm).</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: raid5: Disk failure on sdm, disabling device. Operation continuing on 7 devices</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: raid5:md12: read error not correctable (sector 1948444168 on sdm).</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:34 oss07 kernel: raid5:md12: read error not correctable (sector 1948444176 on sdm).</font></div></div><div><font class="Apple-style-span" face="Menlo">... ...</font></div><div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: --- rd:10 wd:7 fd:3</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 0, o:1, dev:sdu</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 1, o:0, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:49 oss07 kernel: disk 9, o:1, dev:sdw</font></div><div><font class="Apple-style-span" face="Menlo">... ...</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: RAID5 conf printout:</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: --- rd:10 wd:7 fd:3</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 1, o:0, dev:sdm</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 2, o:1, dev:sdn</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 3, o:1, dev:sdo</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 5, o:1, dev:sdq</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 6, o:1, dev:sdr</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 7, o:1, dev:sds</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 8, o:1, dev:sdt</font></div><div><font class="Apple-style-span" face="Menlo">May 7 09:42:58 oss07 kernel: disk 9, o:1, dev:sdw</font></div></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div># current md status</div><div><div>[root@oss07 ~]# mdadm --detail /dev/md12</div><div>/dev/md12:</div><div> Version : 00.90.03</div><div> Creation Time : Mon Oct 4 15:30:53 2010</div><div> Raid Level : raid6</div><div> Array Size : 7814099968 (7452.11 GiB 8001.64 GB)</div><div> Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)</div><div> Raid Devices : 10</div><div> Total Devices : 11</div><div>Preferred Minor : 12</div><div> Persistence : Superblock is persistent</div><div><br></div><div> Intent Bitmap : /mnt/scratch/bitmaps/ost02/bitmap</div><div><br></div><div> Update Time : Mon May 7 11:38:51 2012</div><div> State : clean, degraded</div><div> Active Devices : 7</div><div>Working Devices : 8</div><div> Failed Devices : 3</div><div> Spare Devices : 1</div><div><br></div><div> Chunk Size : 128K</div><div><br></div><div> UUID : 63eb5b15:294c1354:f0c167bd:f8e81f47</div><div> Events : 0.7382</div><div><br></div><div> Number Major Minor RaidDevice State</div><div> 0 0 0 0 removed</div><div> 1 0 0 1 removed</div><div> 2 8 208 2 active sync /dev/sdn</div><div> 3 8 224 3 active sync /dev/sdo</div><div> 4 0 0 4 removed</div><div> 5 65 0 5 active sync /dev/sdq</div><div> 6 65 16 6 active sync /dev/sdr</div><div> 7 65 32 7 active sync /dev/sds</div><div> 8 65 48 8 active sync /dev/sdt</div><div> 9 65 96 9 active sync /dev/sdw</div><div><br></div><div> 10 8 176 - faulty spare /dev/sdl</div><div> 11 65 64 - spare /dev/sdu</div><div> 12 8 240 - faulty spare /dev/sdp</div><div> 13 8 192 - faulty spare /dev/sdm</div></div><div><br></div><div><br></div><div><font class="Apple-style-span" face="Menlo">Best regards,</font></div><div><font class="Apple-style-span" face="Menlo"><br></font></div><div><font class="Apple-style-span" face="Menlo"><div>Taeyoung Hong</div><div>Senior Researcher</div><div><font class="Apple-style-span">Supercomputing Center of KISTI </font></div></font></div></body></html>