<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>We use a simple multipath config and then have our vdev_id.conf set up like the following:</p>
<pre>multipath yes
# Intent of channel names:
# First letter {L,U} indicates lower or upper enclosure
# PCI_ID HBA PORT CHANNEL NAME
channel 05:00.0 1 L
channel 05:00.0 0 U
channel 06:00.0 1 L
channel 06:00.0 0 U
</pre>
This results in the devices being available in /dev/disk/by-vdev/L{1..N} and U{1..N}. We then create the zpools using those L and U devices. When we fail a drive, we use zpool offline -f <[U|L]X> to fault the device (unless the ZED has already faulted it automatically).
When the drive is "faulted" with that -f option, the ZED automatically resilvers the drive when replaced. We never manipulate multipath.<br>
<p><br>
</p>
<div class="moz-cite-prefix">On 4/26/19 9:50 AM, Riccardo Veraldi wrote:<br>
</div>
<blockquote type="cite" cite="mid:efd68d02-e13b-9a4c-82c2-12e71d388ac7@gmail.com">
<div class="moz-cite-prefix">for my experience multipathd+ZFS works well, and it worked well usually.<br>
</div>
<div class="moz-cite-prefix">I just remove the broken disk when it happens, replace it and the new multipathd device is added once the disk is replaced, and then then I start resilvering.</div>
<div class="moz-cite-prefix">Anyway I found out this not always works with some version of JBOD disk array/firmware.</div>
<div class="moz-cite-prefix">Some Proware controller that I had did not recognize that a disk was replaced. But This is not a multipathd problem in my case.</div>
<div class="moz-cite-prefix">So my hint is to try it out with your hardware and see how it behaves.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 26/04/2019 16:57, Kurt Strosahl wrote:<br>
</div>
<blockquote type="cite" cite="mid:BN8PR09MB357083DC99E9212552387B41A73E0@BN8PR09MB3570.namprd09.prod.outlook.com">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hey, thanks!</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I tried the multipathing part you had down there and I couldn't get it to work... I did find that this worked though</p>
<div><br>
#I pick a victim device</div>
<div>multipath -ll</div>
<div>...</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div>#then I remove the device</div>
<div>multipath -f mpathax</div>
<div>#and verify that it is gone</div>
<div>multipath -ll | grep mpathax</div>
<div>#then I run the following, which seems to rescan for devices.</div>
<div>multipath -v2</div>
<div>Apr 26 10:49:06 | sdj: No SAS end device for 'end_device-1:1'</div>
<div>Apr 26 10:49:06 | sddy: No SAS end device for 'end_device-11:1'</div>
<div>create: mpathax (35000cca2680a8194) undef HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=undef</div>
<div>`-+- policy='service-time 0' prio=1 status=undef</div>
<div> |- 1:0:10:0 sdj 8:144 undef ready running</div>
<div> `- 11:0:9:0 sddy 128:0 undef ready running</div>
<div>#then its back</div>
<div>multipath -ll mpathax</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div><br>
</div>
I still need to test it fully once I get the whole stack up and running, but this seems to be a step in the right direction.
<p style="margin-top:0;margin-bottom:0"><br>
</p>
w/r,
<div>Kurt<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jongwoo Han
<a class="moz-txt-link-rfc2396E" href="mailto:jongwoohan@gmail.com" moz-do-not-send="true">
<jongwoohan@gmail.com></a><br>
<b>Sent:</b> Friday, April 26, 2019 6:28 AM<br>
<b>To:</b> Kurt Strosahl<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" moz-do-not-send="true">
lustre-discuss@lists.lustre.org</a><br>
<b>Subject:</b> Re: [lustre-discuss] ZFS and multipathing for OSTs</font>
<div> </div>
</div>
<div>
<div dir="ltr">Disk replacement with multipathd + zfs is somewhat not convenient.
<div><br>
</div>
<div>step1: mark offline the disk you should replace with zpool command</div>
<div>step2: remove disk from multipathd table with multipath -f <mpath id></div>
<div>step3: replace disk</div>
<div>step4: add disk to multipath table with multipath -ll <mpath id></div>
<div>step5: replace disk in zpool with zpool replace</div>
<div><br>
</div>
<div>try this in your test environment and tell us if you have found anything interesting in the syslog.</div>
<div>In my case replacing single disk in multipathd+zfs pool triggerd massive udevd partition scan. </div>
<div><br>
</div>
<div>Thanks</div>
<div>Jongwoo Han</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">2019년 4월 26일 (금) 오전 3:44, Kurt Strosahl <<a href="mailto:strosahl@jlab.org" id="LPlnk629042" class="OWAAutoLink" previewremoved="true" moz-do-not-send="true">strosahl@jlab.org</a>>님이 작성:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px
0px 0px 0.8ex; border-left:1px solid
rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div id="x_gmail-m_-296173503153998559divtagdefaultwrapper" dir="ltr" style="font-size:12pt;
color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color
Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android
Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">Good Afternoon,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"> As part of a new lustre deployment I've now got two disk shelves connected redundantly to two servers. Since each disk has two paths to the server I'd like to use multipathing for both redundancy and improved
performance. I haven't found examples or discussion about such a setup, and was wondering if there are any resources out there that I could consult.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Of particular interest would be examples of the /etc/zfs/vdev_id.conf and any tuning that was done. I'm also wondering about extra steps that may have to be taken when doing a disk replacement to account for the
multipathing. I've got plenty of time to experiment with this process, but I'd rather not reinvent the wheel if I don't have to.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">w/r,</p>
<div id="x_gmail-m_-296173503153998559Signature">
<div id="x_gmail-m_-296173503153998559divtagdefaultwrapper" dir="ltr" style="font-size:12pt;
color:rgb(0,0,0);
font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple
Color Emoji","Segoe UI
Emoji",NotoColorEmoji,"Segoe UI
Symbol","Android
Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span id="x_gmail-m_-296173503153998559ms-rterangepaste-start"></span><span style="color:rgb(51,51,51);
font-family:monospace;
font-size:14.16px">Kurt
J. Strosahl</span><br style="color:rgb(51,51,51);
font-family:monospace;
font-size:14.16px">
<span style="color:rgb(51,51,51);
font-family:monospace;
font-size:14.16px">System Administrator: Lustre, HPC</span><br style="color:rgb(51,51,51);
font-family:monospace;
font-size:14.16px">
<span style="color:rgb(51,51,51);
font-family:monospace;
font-size:14.16px">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="x_gmail-m_-296173503153998559ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk586564" class="OWAAutoLink" previewremoved="true" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br>
<a href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=02%7C01%7Cstrosahl%40jlab.org%7Cba16f1aff6144708f17708d6ca31e3ee%7Cb4d7ee1f4fb34f0690372b5b522042ab%7C1%7C1%7C636918712958511376&sdata=p6QC1JIfSnyq8IC1SgOJWlWdcD2Drs9vbtrutuynGEs%3D&reserved=0" originalsrc="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" shash="Pd4jexId8Z0d/E64s9fxAY1wf+F1P7cO9+YvuZMwmvFaEwKNmC5uC0D7sDQ+hiB8R9fe6SGnJ6kev+2IujBkMOiEK+gO/V5sEFKS1AT8Q4VA0VE7jCnxCP+shJ9KoOltZLRsrhmvPx/LHUbsanZslHVXbyhwPhFp2eDd2IeJdKo=" rel="noreferrer" target="_blank" id="LPlnk520695" class="OWAAutoLink" previewremoved="true" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature"><font face="'courier new', monospace">Jongwoo Han</font>
<div><font face="'courier new', monospace">+82-505-227-6108</font></div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org" moz-do-not-send="true">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
<p><br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
lustre-discuss mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a>
</pre>
</blockquote>
</body>
</html>