<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Patrick didn\u2019t say it, but in case it wan\u2019t obvious you could us MDRAID underneath ldiskfs to achieve redundancy under a single host.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Moreover, if you do, then you can have larger OSTs, which is helpful for file system usability. You\u2019ll have fewer OSTs to manage and they will be larger, which prevents them from filing as easy. It can be a pain to deal with full OSTs
and you can avoid the ENOSPC or the hassle of rebalancing them if you assemble larger OSTs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-Cory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Patrick Farrell <paf@cray.com><br>
<b>Date: </b>Wednesday, June 27, 2018 at 8:29 AM<br>
<b>To: </b>yu sun <sunyu1949@gmail.com>, "adilger@whamcloud.com" <adilger@whamcloud.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] lctl ping node28@o2ib report Input/output error<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<br>
I\u2019m a little puzzled - it can switch, but isn\u2019t the data on the failed disk lost...? That\u2019s why Andreas is suggesting RAID. Or is drbd doing syncing of the disk? That seems like a really expensive way to get redundancy, since it would have to be full online
mirroring with all the costs in hardware and resource usage that implies...?<br>
<br>
ZFS is not a requirement, it generally performs a bit worse than ldiskfs but makes it up with impressive features to improve data integrity and related things. Since it sounds like that\u2019s not a huge concern for you, I would stick with ldiskfs. It will likely
be a little faster and is easier to set up.<o:p></o:p></p>
<div class="MsoNormal" align="center" style="margin-left:.5in;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="color:black">From:</span></b><span style="color:black"> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of yu sun <sunyu1949@gmail.com><br>
<b>Sent:</b> Wednesday, June 27, 2018 8:21:43 AM<br>
<b>To:</b> adilger@whamcloud.com<br>
<b>Cc:</b> lustre-discuss@lists.lustre.org<br>
<b>Subject:</b> Re: [lustre-discuss] lctl ping node28@o2ib report Input/output error</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">yes<span style="font-family:"MS Gothic"">\uff0c</span> you are right, thanks for your great suggestions.
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in">now we are using glusterfs to store training data for ML, and we begin to investigate lustre to instead glusterfs for performance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Firstly, yes we do want to get maximum perforance, you means we should use zfs , for example , not each ost/mdt on a separate partitions, for better perforance?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Secondly, we dont use any underlying RAID devices, and we do configure each ost on a separate disk, considering that lustre does not provide disk data redundancy, we are use drbd + pacemarker + corosync for data
<span style="background:white">redundancy and HA, you can see we have configured --servicenode when mkfs.lustre. I dont know how reliable is this solution? it seems ok for our current test, when one disk faild, pacemarker can switch to other ost on the other
machine </span>automaticly.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="background:white"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="background:white">we also want to use zfs and I have test zfs by mirror, However, if the physical machine down</span><span style="font-family:"MS Gothic";background:white">\uff0c</span><span style="background:white">data
on the machine will lost. so we decice use the solution listed above.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Now we are testing, and any suggesting is appreciated
<span style="font-family:"Apple Color Emoji"">😆</span>.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">thanks <span style="background:white">
Andreas.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="background:white"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="background:white">Your</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="background:white">Yu</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Andreas Dilger <<a href="mailto:adilger@whamcloud.com">adilger@whamcloud.com</a>>
<span style="font-family:"MS Gothic"">\u4e8e</span>2018<span style="font-family:"MS Gothic"">\u5e74</span>6<span style="font-family:"MS Gothic"">\u6708</span>27<span style="font-family:"MS Gothic"">\u65e5\u5468\u4e09</span>
<span style="font-family:"MS Gothic"">\u4e0b\u5348</span>7:07<span style="font-family:"MS Gothic"">\u5199\u9053\uff1a</span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
On Jun 27, 2018, at 09:12, yu sun <<a href="mailto:sunyu1949@gmail.com" target="_blank">sunyu1949@gmail.com</a>> wrote:<br>
> <br>
> client:<br>
> root@ml-gpu-ser200.nmg01:~$ mount -t lustre node28@o2ib1:node29@o2ib1:/project /mnt/lustre_data<br>
> mount.lustre: mount node28@o2ib1:node29@o2ib1:/project at /mnt/lustre_data failed: Input/output error<br>
> Is the MGS running?<br>
> root@ml-gpu-ser200.nmg01:~$ lctl ping node28@o2ib1<br>
> failed to ping 10.82.143.202@o2ib1: Input/output error<br>
> root@ml-gpu-ser200.nmg01:~$<br>
> <br>
> <br>
> mgs and mds:<br>
> mkfs.lustre --mgs --reformat --servicenode=node28@o2ib1 --servicenode=node29@o2ib1 /dev/sdb1<br>
> mkfs.lustre --fsname=project --mdt --index=0 --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode node28@o2ib1 --servicenode node29@o2ib1 --reformat --backfstype=ldiskfs /dev/sdc1<br>
<br>
Separate from the LNet issues, it is probably worthwhile to point out some issues<br>
with your configuration. You shouldn't use partitions on the OST and MDT devices<br>
if you want to get maximum performance. That can offset all of the filesystem IO<br>
from the RAID/sector alignment and hurt performance.<br>
<br>
Secondly, it isn't clear if you are using underlying RAID devices, or if you are<br>
configuring each OST on a separate disk? It looks like the latter - that you are<br>
making each disk a separate OST. That isn't a good idea for Lustre, since it does<br>
not (yet) have any redundancy at higher layers, and any disk failure would result<br>
in data loss. You currently need to have RAID-5/6 or ZFS for each OST/MDT, unless<br>
this is a really "scratch" filesystem where you don't care if the data is lost and<br>
reformatting the filesystem is OK (i.e. low cost is the primary goal, which is fine<br>
also, but not very common).<br>
<br>
We are working at Lustre-level data redundancy, and there is some support for this<br>
in the 2.11 release, but it is not yet in a state where you could reliably use it<br>
to mirror all of the files in the filesystem.<br>
<br>
Cheers, Andreas<br>
<br>
> <br>
> ost:<br>
> ml-storage-ser22.nmg01:<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=12 /dev/sdc1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=13 /dev/sdd1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=14 /dev/sde1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=15 /dev/sdf1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=16 /dev/sdg1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=17 /dev/sdh1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=18 /dev/sdi1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=19 /dev/sdj1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=20 /dev/sdk1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=21 /dev/sdl1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=22 /dev/sdm1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node22@o2ib1 --servicenode=node23@o2ib1 --ost --index=23 /dev/sdn1<br>
> ml-storage-ser26.nmg01:<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=36 /dev/sdc1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=37 /dev/sdd1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=38 /dev/sde1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=39 /dev/sdf1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=40 /dev/sdg1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=41 /dev/sdh1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=42 /dev/sdi1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=43 /dev/sdj1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=44 /dev/sdk1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=45 /dev/sdl1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=46 /dev/sdm1<br>
> mkfs.lustre --fsname=project --reformat --mgsnode=node28@o2ib1 --mgsnode=node29@o2ib1 --servicenode=node26@o2ib1 --servicenode=node27@o2ib1 --ost --index=47 /dev/sdn1<br>
> <br>
> Thanks<br>
> Yu<br>
> <br>
> Mohr Jr, Richard Frank (Rick Mohr) <<a href="mailto:rmohr@utk.edu" target="_blank">rmohr@utk.edu</a>>
<span style="font-family:"MS Gothic"">\u4e8e</span>2018<span style="font-family:"MS Gothic"">\u5e74</span>6<span style="font-family:"MS Gothic"">\u6708</span>27<span style="font-family:"MS Gothic"">\u65e5\u5468\u4e09</span>
<span style="font-family:"MS Gothic"">\u4e0b\u5348</span>1:25<span style="font-family:"MS Gothic"">\u5199\u9053\uff1a</span><br>
> <br>
> > On Jun 27, 2018, at 12:52 AM, yu sun <<a href="mailto:sunyu1949@gmail.com" target="_blank">sunyu1949@gmail.com</a>> wrote:<br>
> ><br>
> > I have create file /etc/modprobe.d/lustre.conf with content on all mdt ost and client:<br>
> > root@ml-gpu-ser200.nmg01:~$ cat /etc/modprobe.d/lustre.conf<br>
> > options lnet networks="o2ib1(eth3.2)"<br>
> > and I exec command line : lnetctl lnet configure --all to make my static lnet configuration take effect. but i still can't ping node28 from my client ml-gpu-ser200.nmg01. I can mount as well as access lustre on client ml-gpu-ser200.nmg01.<br>
> <br>
> What options did you use when mounting the file system?<br>
> <br>
> --<br>
> Rick Mohr<br>
> Senior HPC System Administrator<br>
> National Institute for Computational Sciences<br>
> <a href="http://www.nics.tennessee.edu" target="_blank">http://www.nics.tennessee.edu</a><br>
> <br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" target="_blank">
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
<br>
Cheers, Andreas<br>
---<br>
Andreas Dilger<br>
Principal Lustre Architect<br>
Whamcloud<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>