<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ks_c_5601-1987">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hey, thanks!</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I tried the multipathing part you had down there and I couldn't get it to work... I did find that this worked though</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div><br>
#I pick a victim device</div>
<div></div>
<div>multipath -ll</div>
<div>...</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div></div>
<div>#then I remove the device</div>
<div>multipath -f mpathax</div>
<div>#and verify that it is gone</div>
<div>multipath -ll | grep mpathax</div>
<div></div>
<div>#then I run the following, which seems to rescan for devices.</div>
<div>multipath -v2</div>
<div>Apr 26 10:49:06 | sdj: No SAS end device for 'end_device-1:1'</div>
<div>Apr 26 10:49:06 | sddy: No SAS end device for 'end_device-11:1'</div>
<div>create: mpathax (35000cca2680a8194) undef HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=undef</div>
<div>`-+- policy='service-time 0' prio=1 status=undef</div>
<div> |- 1:0:10:0 sdj 8:144 undef ready running</div>
<div> `- 11:0:9:0 sddy 128:0 undef ready running</div>
<div></div>
<div>#then its back</div>
<div>multipath -ll mpathax</div>
<div>mpathax (35000cca2680a8194) dm-49 HGST ,HUH721010AL5200 </div>
<div>size=9.1T features='0' hwhandler='0' wp=rw</div>
<div>`-+- policy='service-time 0' prio=1 status=enabled</div>
<div> |- 1:0:10:0 sdj 8:144 active ready running</div>
<div> `- 11:0:9:0 sddy 128:0 active ready running</div>
<div><br>
</div>
I still need to test it fully once I get the whole stack up and running, but this seems to be a step in the right direction.
<p></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
w/r,
<div>Kurt<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Jongwoo Han <jongwoohan@gmail.com><br>
<b>Sent:</b> Friday, April 26, 2019 6:28 AM<br>
<b>To:</b> Kurt Strosahl<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] ZFS and multipathing for OSTs</font>
<div> </div>
</div>
<div>
<div dir="ltr">Disk replacement with multipathd + zfs is somewhat not convenient.
<div><br>
</div>
<div>step1: mark offline the disk you should replace with zpool command</div>
<div>step2: remove disk from multipathd table with multipath -f <mpath id></div>
<div>step3: replace disk</div>
<div>step4: add disk to multipath table with multipath -ll <mpath id></div>
<div>step5: replace disk in zpool with zpool replace</div>
<div><br>
</div>
<div>try this in your test environment and tell us if you have found anything interesting in the syslog.</div>
<div>In my case replacing single disk in multipathd+zfs pool triggerd massive udevd partition scan. </div>
<div><br>
</div>
<div>Thanks</div>
<div>Jongwoo Han</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">2019 4 26 () 3:44, Kurt Strosahl <<a href="mailto:strosahl@jlab.org" id="LPlnk629042" class="OWAAutoLink" previewremoved="true">strosahl@jlab.org</a>> \u06fc:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div id="x_gmail-m_-296173503153998559divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px">Good Afternoon,</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px"> As part of a new lustre deployment I've now got two disk shelves connected redundantly to two servers. Since each disk has two paths to the server I'd like to use multipathing for both redundancy and improved
performance. I haven't found examples or discussion about such a setup, and was wondering if there are any resources out there that I could consult.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">Of particular interest would be examples of the /etc/zfs/vdev_id.conf and any tuning that was done. I'm also wondering about extra steps that may have to be taken when doing a disk replacement to account for the
multipathing. I've got plenty of time to experiment with this process, but I'd rather not reinvent the wheel if I don't have to.</p>
<p style="margin-top:0px; margin-bottom:0px"><br>
</p>
<p style="margin-top:0px; margin-bottom:0px">w/r,</p>
<div id="x_gmail-m_-296173503153998559Signature">
<div id="x_gmail-m_-296173503153998559divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px; margin-bottom:0px"><span id="x_gmail-m_-296173503153998559ms-rterangepaste-start"></span><span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">Kurt J. Strosahl</span><br style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">System Administrator: Lustre, HPC</span><br style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">
<span style="color:rgb(51,51,51); font-family:monospace; font-size:14.16px">Scientific Computing Group, Thomas Jefferson National Accelerator Facility</span><span id="x_gmail-m_-296173503153998559ms-rterangepaste-end"></span><br>
</p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank" id="LPlnk586564" class="OWAAutoLink" previewremoved="true">lustre-discuss@lists.lustre.org</a><br>
<a href="https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-discuss-lustre.org&data=02%7C01%7Cstrosahl%40jlab.org%7Cba16f1aff6144708f17708d6ca31e3ee%7Cb4d7ee1f4fb34f0690372b5b522042ab%7C1%7C1%7C636918712958511376&sdata=p6QC1JIfSnyq8IC1SgOJWlWdcD2Drs9vbtrutuynGEs%3D&reserved=0" originalsrc="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" shash="Pd4jexId8Z0d/E64s9fxAY1wf+F1P7cO9+YvuZMwmvFaEwKNmC5uC0D7sDQ+hiB8R9fe6SGnJ6kev+2IujBkMOiEK+gO/V5sEFKS1AT8Q4VA0VE7jCnxCP+shJ9KoOltZLRsrhmvPx/LHUbsanZslHVXbyhwPhFp2eDd2IeJdKo=" rel="noreferrer" target="_blank" id="LPlnk520695" class="OWAAutoLink" previewremoved="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature"><font face="'courier new', monospace">Jongwoo Han</font>
<div><font face="'courier new', monospace">+82-505-227-6108</font></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>