<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello, thanks for this - it's a 16 disk raid10 (with one spare) so 24Tb. I previously tried 1.0 and 1.2 metadata, to no effect.<div>we are using 256 chunk sizes, I haven't tried reverting to 64k but will do so.</div><div><br></div><div>this looks to me like something different - there was an md patch for 2.6.18 kernels relating to updating of superblocks on arrays,</div><div>which this might be ...</div><div><br></div><div>the storage vendor has tried md with 3Tb disks and md version 3.2.3 and sees no problems, but this is also with a much later kernel version.</div><div>I am not sure 3.2.3 would work with kernels as old as 2.6.18-274</div><div><br></div><div><div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Professor Samuel Aparicio BM BCh PhD FRCPath<br>Nan and Lorraine Robertson Chair UBC/BC Cancer Agency<br>675 West 10th, Vancouver V5Z 1L3, Canada.<br>office: +1 604 675 8200 lab website <a href="http://molonc.bccrc.ca">http://molonc.bccrc.ca</a><br><br><br></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br><div><div>On Mar 19, 2012, at 11:46 PM, Robin Humble wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On Mon, Mar 19, 2012 at 10:05:42PM -0700, Samuel Aparicio wrote:<br><blockquote type="cite">I am wondering if anyone has experienced issues with md / mdadm in the 1.8.7-wc1 patched server kernels.?<br></blockquote><br>I've seen an issue:<br> <a href="http://jira.whamcloud.com/browse/LU-1115">http://jira.whamcloud.com/browse/LU-1115</a><br>althought it looks quite different to your problem... still, you might<br>hit that problem next.<br><br><blockquote type="cite">we have historically used software raid on our OSS machines because it provided a 20-30% throughput in our hands, over<br></blockquote><blockquote type="cite">raid provided from our storage arrays (coraid ATA over ethernet shelves). In 1.8.5 this has worked more or less flawlessly,<br></blockquote><blockquote type="cite">but we now have new storage, with 3Tb rather than 2Tb disks and new servers with 1.8.7-wc1 patched kernels.<br></blockquote><br>I don't have any 3tb disks to test with, but I think you need to use a<br>newer superblock format for 3tb devices.<br>eg. use<br> mdadm -e 1.2 ...<br>see 'man mdadm' which says something about max 2tb devices for 0.90 format.<br><br>also I'm not quite sure how to read the below, but it kinda looks like<br>you have 17 3tb disks in a single raid? that's a lot... I thought<br>ldiskfs was only ok up to 24tb these days?<br><br><blockquote type="cite">md is unable to reliably shut down and restart arrays after the machines have been rebooted (cleanly) - the disks are no<br></blockquote><blockquote type="cite">longer recognized as part of the arrays they were created within. In the kernel log we have seen the following messages below,<br></blockquote><blockquote type="cite">which include the following:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">md: bug in file drivers/md/md.c, line 1677<br></blockquote><br> if (!mddev->events) {<br> /*<br> * oops, this 64-bit counter should never wrap.<br> * Either we are in around ~1 trillion A.C., assuming<br> * 1 reboot per second, or we have a bug:<br> */<br> MD_BUG();<br> mddev->events --;<br> }<br><br><br>so it looks like your md superblock is corrupted. that's consistent with<br>needing a newer superblock version.<br><br>other less likely possibilities:<br> - could it also be that your coraid devices have problems with >2TB?<br> - if you are running with 32bit kernels something could be wrong there.<br><br>cheers,<br>robin<br>--<br>Dr Robin Humble, HPC Systems Analyst, NCI National Facility<br><br><blockquote type="cite">looking through the mdadm changelogs, it seems like there are some possible patches for md in 2.6.18 kernels but I cannot tell<br></blockquote><blockquote type="cite">if they are applied here, or whether this is even relevant.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I am not clear whether this is an issue with 3Tb disks, or something else related to mdadm and the patched server kernel. My suspicion<br></blockquote><blockquote type="cite">is that something has broken with > 2.2Tb disks.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Does anyone have any ideas about this?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">thanks<br></blockquote><blockquote type="cite">sam aparicio<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">---------------<br></blockquote><blockquote type="cite">Mar 19 21:34:48 OST3 kernel: md: **********************************<br></blockquote><blockquote type="cite">Mar 19 21:34:48 OST3 kernel: <br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: bug in file drivers/md/md.c, line 1677<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: <br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: **********************************<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: * <COMPLETE RAID STATE PRINTOUT> *<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: **********************************<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md142: <br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md141: <br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md140: <etherd/e14.16><etherd/e14.15><etherd/e14.14><etherd/e14.13><etherd/e14.12><etherd/e14.11><etherd/e14.10><etherd/e14.9><etherd/e14.8><etherd/e14.7><etherd/e14.6><etherd<br></blockquote><blockquote type="cite">/e14.5><etherd/e14.4><etherd/e14.3><etherd/e14.2><etherd/e14.1><etherd/e14.0><br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: rdev etherd/e14.16, SZ:2930265344 F:0 S:0 DN:16<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: rdev superblock:<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: SB: (V:1.0.0) ID:<9859f274.34313a61.00000030.00000000> CT:5d3314af<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: L234772919 S861164367 ND:1970037550 RD:1919251571 md1667457582 LO:65536 CS:196610<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: UT:00000800 ST:0 AD:1565563648 WD:1 FD:8 SD:0 CSUM:00000000 E:00000000<br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: D 0: DISK<N:-1,(-1,-1),R:-1,S:-1><br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: D 1: DISK<N:-1,(-1,-1),R:-1,S:-1><br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: D 2: DISK<N:-1,(-1,-1),R:-1,S:-1><br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: D 3: DISK<N:-1,(-1,-1),R:-1,S:-1><br></blockquote><blockquote type="cite">Mar 19 21:35:20 OST3 kernel: md: THIS: DISK<N:0,(0,0),R:0,S:0><br></blockquote><blockquote type="cite">< output truncated ><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Professor Samuel Aparicio BM BCh PhD FRCPath<br></blockquote><blockquote type="cite">Nan and Lorraine Robertson Chair UBC/BC Cancer Agency<br></blockquote><blockquote type="cite">675 West 10th, Vancouver V5Z 1L3, Canada.<br></blockquote><blockquote type="cite">office: +1 604 675 8200 lab website <a href="http://molonc.bccrc.ca">http://molonc.bccrc.ca</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Lustre-discuss mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br></blockquote><blockquote type="cite"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></blockquote></div></blockquote></div><br></div></div></body></html>