<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
Date: Mon, 5 Oct 2009 15:42:25 +0100<br>
From: <a href="mailto:pg_lus@lus.for.sabi.co.UK">pg_lus@lus.for.sabi.co.UK</a> (Peter Grandi)<br>
Subject: Re: [Lustre-discuss] drbd slow I/O with lustre filesystem<br>
To: Lustre discussion <<a href="mailto:lustre-discuss@lists.Lustre.org">lustre-discuss@lists.Lustre.org</a>><br>
Message-ID: <<a href="mailto:19146.1489.532204.6707@tree.ty.sabi.co.uk">19146.1489.532204.6707@tree.ty.sabi.co.uk</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
</blockquote><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
RAID5 over RAID1? Nahh. Consider <a href="http://WWW.BAARF.com/" target="_blank">http://WWW.BAARF.com/</a> and that<br>
the storage system of a Lustre pool over DRBD is ideally suited to<br>
RAID10 (with each pair a DRBD resource). RAID5 may be contributing<br>
to your speed problem below because of or being rebuilt/syncing<br>
itself.<br>
<br></blockquote><div>Poor me, i don't know it before, so now we can't change anything on my raid partition :( . <br> <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> After formatting them with lustre format ( using mkfs.lustre ) ,<br>
> i start to copy data to my drbd devices, but:<br>
<br>
> - Its I/O wait when i monitor by top or iostat is too hight,<br>
> about 25%<br>
<br>
This is not much related to anything... After all you are doing a<br>
lot of IO, and jumping around on the disk, doing a restore.<br></blockquote><div><br>Could you please tell me in detail what do you mean is ? I don't really understand it ? <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
> - The copy speed from my web client to our OST using drbd<br>
> devices is too low, only about 13MB/s although client and ost in<br>
> is the same 1Gb Ethernet LAN.<br>
<br>
Too few details about this. Thigns to check:<br>
<br>
* Raw network speed: I like 'nuttcp' to do check it. Using the<br>
usual trick (larger send/receive buffers, jumbo frames, ...) may<br>
help if there are issues. But then you were getting 70MB/s above.<br>
<a href="http://lists.centos.org/pipermail/centos/2009-July/079505.html" target="_blank">http://lists.centos.org/pipermail/centos/2009-July/079505.html</a><br>
* If you are using LVMN2 bad news.<br>
<a href="http://archives.free.net.ph/message/20070815.091608.fff62ba9.en.html" target="_blank">http://archives.free.net.ph/message/20070815.091608.fff62ba9.en.html</a><br>
* Using RAID5 as argued above may be detrimental.<br>
* The DRBD must be configured to allow higher sync speeds:<br>
<a href="http://www.ossramblings.com/drbd_defaults_too_slow" target="_blank">http://www.ossramblings.com/drbd_defaults_too_slow</a><br>
<a href="http://www.linux-ha.org/DRBD/FAQ#head-e09d2c15ba7ff691ecd5d5d7b848a50d25a3c3eb" target="_blank">http://www.linux-ha.org/DRBD/FAQ#head-e09d2c15ba7ff691ecd5d5d7b848a50d25a3c3eb</a><br>
Your initial sync however seemed to run at 70MB/s so<br>
I wonder. Maybe tuning the "unplug" waterkmark in DRBD<br>
or if you have battery backup enabling no-flush mode.<br>
<a href="http://archives.free.net.ph/message/20081219.085301.997727d2.en.html" target="_blank">http://archives.free.net.ph/message/20081219.085301.997727d2.en.html</a><br>
<br>
> When i tried using one OST without drbd, it worked quite well<br>
<br>
It might mean that it is mainly a DRBD issue. You might want to<br>
get the latest DRBD versions, as some earlier versions. If you<br>
have RHEL the ElRepo has got fairly recent ones.<br>
<br>
> So, could any one please tell me where the problem is ? In our<br>
> drbd devices or because of lustre ? Is there anyone has the same<br>
> problem with me ? :(<br>
<br>
All of the above probably -- max performance here means ensuring<br>
that write requests are issued as fast as possible and back-to-back<br>
packets/blocks are then possible both on the network and on the<br>
storage system...<br>
<br>
<a href="http://www.gossamer-threads.com/lists/drbd/users/17991" target="_blank">http://www.gossamer-threads.com/lists/drbd/users/17991</a><br>
<a href="http://lists.linbit.com/pipermail/drbd-user/2007-August/007256.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2007-August/007256.html</a><br>
<a href="http://lists.linbit.com/pipermail/drbd-user/2009-January/011165.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2009-January/011165.html</a><br>
<a href="http://lists.linbit.com/pipermail/drbd-user/2009-January/011198.html" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2009-January/011198.html</a><br></blockquote><div><br>They are really great information, i checked it but will consider to using some of them ( i.e some drbd options like no-disk-flushes, no-md-flushes ... it's maybe useful in speed tuning but i am not sure it won't affect to my system stability ) <br>
<br>Anyway, many thanks for all of them :) <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
It may conceivably be quicker for you to load all your data first<br>
on the primary storage half of the pair, and then reactivate the<br>
secondary and let resync.<br>
<br></blockquote><div>I tried using that way but the speed increasing is not remarkable, about 5-7MB <br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
My impression is that a problem is unlikely to originate in the<br>
Lustre side, but more on the underlying layers mentioned above.<br>
There is a fair bit of material on DRBD optimization, both on its<br>
site, and more specifically around the MySQL community, where it<br>
is very commonly used, and they care a lot about performance.<br>
<br>
<br></blockquote><div>It's also what i guessed, so i posted my questions to both lustre and drbd mailing list and luckily, i received some useful information and tips. <br><br>After all, many thanks for you detail answer. I'm really appreciated it :) <br>
<input id="gwProxy" type="hidden"><input onclick="jsCall();" id="jsProxy" type="hidden"></div><br></div><br><div id="refHTML"></div>