<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">for my experience multipathd+ZFS works
well, and it worked well usually.<br>
</div>
<div class="moz-cite-prefix">I just remove the broken disk when it
happens, replace it and the new multipathd device is added once
the disk is replaced, and then then I start resilvering.</div>
<div class="moz-cite-prefix">Anyway I found out this not always
works with some version of JBOD disk array/firmware.</div>
<div class="moz-cite-prefix">Some Proware controller that I had did
not recognize that a disk was replaced. But This is not a
multipathd problem in my case.</div>
<div class="moz-cite-prefix">So my hint is to try it out with your
hardware and see how it behaves.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 26/04/2019 16:57, Kurt Strosahl
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN8PR09MB357083DC99E9212552387B41A73E0@BN8PR09MB3570.namprd09.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hey, thanks!</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I tried the multipathing
part you had down there and I couldn't get it to work... I did
find that this worked though</p>
<div><br>
#I pick a victim device</div>
<div>multipath -ll</div>
<div>...</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div>#then I remove the device</div>
<div>multipath -f mpathax</div>
<div>#and verify that it is gone</div>
<div>multipath -ll | grep mpathax</div>
<div>#then I run the following, which seems to rescan for
devices.</div>
<div>multipath -v2</div>
<div>Apr 26 10:49:06 | sdj: No SAS end device for
'end_device-1:1'</div>
<div>Apr 26 10:49:06 | sddy: No SAS end device for
'end_device-11:1'</div>
<div>create: mpathax (35000cca2680a8194) undef HGST
,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=undef</div>
<div>`-+- policy='service-time 0' prio=1 status=undef</div>
<div> |- 1:0:10:0 sdj 8:144 undef ready running</div>
<div> `- 11:0:9:0 sddy 128:0 undef ready running</div>
<div>#then its back</div>
<div>multipath -ll mpathax</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div><br>
</div>
I still need to test it fully once I get the whole stack up and
running, but this seems to be a step in the right direction.
<p style="margin-top:0;margin-bottom:0"><br>
</p>
w/r,
<div>Kurt<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> Jongwoo Han
<a class="moz-txt-link-rfc2396E" href="mailto:jongwoohan@gmail.com"><jongwoohan@gmail.com></a><br>
<b>Sent:</b> Friday, April 26, 2019 6:28 AM<br>
<b>To:</b> Kurt Strosahl<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] ZFS and
multipathing for OSTs</font>
<div> </div>
</div>
<div>
<div dir="ltr">Disk replacement with multipathd + zfs is
somewhat not convenient.
<div><br>
</div>
<div>step1: mark offline the disk you should replace
with zpool command</div>
<div>step2: remove disk from multipathd table with
multipath -f <mpath id></div>
<div>step3: replace disk</div>
<div>step4: add disk to multipath table with multipath
-ll <mpath id></div>
<div>step5: replace disk in zpool with zpool replace</div>
<div><br>
</div>
<div>try this in your test environment and tell us if
you have found anything interesting in the syslog.</div>
<div>In my case replacing single disk in multipathd+zfs
pool triggerd massive udevd partition scan. </div>
<div><br>
</div>
<div>Thanks</div>
<div>Jongwoo Han</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">2019년 4월 26일 (금) 오전
3:44, Kurt Strosahl <<a
href="mailto:strosahl@jlab.org" id="LPlnk629042"
class="OWAAutoLink" previewremoved="true"
moz-do-not-send="true">strosahl@jlab.org</a>>님이
작성:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px
0px 0.8ex; border-left:1px solid rgb(204,204,204);
padding-left:1ex">
<div dir="ltr">
<div
id="x_gmail-m_-296173503153998559divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color
Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android
Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">Good
Afternoon,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">
As part of a new lustre deployment I've now got
two disk shelves connected redundantly to two
servers. Since each disk has two paths to the
server I'd like to use multipathing for both
redundancy and improved performance. I haven't
found examples or discussion about such a setup,
and was wondering if there are any resources out
there that I could consult.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Of
particular interest would be examples of the
/etc/zfs/vdev_id.conf and any tuning that was
done. I'm also wondering about extra steps that
may have to be taken when doing a disk
replacement to account for the multipathing.
I've got plenty of time to experiment with this
process, but I'd rather not reinvent the wheel
if I don't have to.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">w/r,</p>
<div id="x_gmail-m_-296173503153998559Signature">
<div
id="x_gmail-m_-296173503153998559divtagdefaultwrapper"
dir="ltr" style="font-size:12pt;
color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android
Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span
id="x_gmail-m_-296173503153998559ms-rterangepaste-start"></span><span
style="color:rgb(51,51,51);
font-family:monospace; font-size:14.16px">Kurt
J. Strosahl</span><br
style="color:rgb(51,51,51);
font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51);
font-family:monospace; font-size:14.16px">System
Administrator: Lustre, HPC</span><br
style="color:rgb(51,51,51);
font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51);
font-family:monospace; font-size:14.16px">Scientific
Computing Group, Thomas Jefferson National
Accelerator Facility</span><span
id="x_gmail-m_-296173503153998559ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org"
target="_blank" id="LPlnk586564" class="OWAAutoLink"
previewremoved="true" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a
href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=02%7C01%7Cstrosahl%40jlab.org%7Cba16f1aff6144708f17708d6ca31e3ee%7Cb4d7ee1f4fb34f0690372b5b522042ab%7C1%7C1%7C636918712958511376&sdata=p6QC1JIfSnyq8IC1SgOJWlWdcD2Drs9vbtrutuynGEs%3D&reserved=0"
originalsrc="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
shash="Pd4jexId8Z0d/E64s9fxAY1wf+F1P7cO9+YvuZMwmvFaEwKNmC5uC0D7sDQ+hiB8R9fe6SGnJ6kev+2IujBkMOiEK+gO/V5sEFKS1AT8Q4VA0VE7jCnxCP+shJ9KoOltZLRsrhmvPx/LHUbsanZslHVXbyhwPhFp2eDd2IeJdKo="
rel="noreferrer" target="_blank" id="LPlnk520695"
class="OWAAutoLink" previewremoved="true"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature"><font
face="'courier new', monospace">Jongwoo Han</font>
<div><font face="'courier new', monospace">+82-505-227-6108</font></div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<p><br>
</p>
</body>
</html>