Hi guys, do you have any idea about my issue, my question? <br><br><div class="gmail_quote">On Wed, Jan 27, 2010 at 8:59 AM, Lex <span dir="ltr"><<a href="mailto:lexluthor87@gmail.com">lexluthor87@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Hi all<br><br>I heard somewhere about oversubscribing issue related to ost thread, but i just wonder why i calculated followed the function that i founded in the manual ( <b>thread_number = RAM * CPU core / 128 MB</b> - do correct me if there's something wrong with it, please ) , the oversubscribing warning is still appears. <br>
<br>Maybe i have to choose my own value from trial and error, but is there any explanation for this situation? <br><br>@Erik : could u please describe your bottleneck problem with journal
device for me ? as detail as better ? <br><div><div></div><div class="h5"><br><br><div class="gmail_quote">On Tue, Jan 26, 2010 at 10:00 PM, Erik Froese <span dir="ltr"><<a href="mailto:erik.froese@gmail.com" target="_blank">erik.froese@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Sorry Lex I misread your email. I saw similar messages about my journal devices. The OST is an ext3+extra features filesystem. Each FS has an associated journal that CAN be on a separate device. Its supposed to speed up small file operations. Mine were oversubscribed and became a bottleneck.<div>
<br></div><div><font color="#888888">Erik</font><div><div></div><div><br><br><div class="gmail_quote">On Mon, Jan 25, 2010 at 11:40 AM, Lex <span dir="ltr"><<a href="mailto:lexluthor87@gmail.com" target="_blank">lexluthor87@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Sorry Erik if i'm rising such a "bad" question, could u tell me more about OST journal device ? I even don't know what it is as well as haven't seen it before, in the lustre manual.<br><br>Best regards <br>
<div><div></div><div>
<br><br><div class="gmail_quote">On Mon, Jan 25, 2010 at 10:52 PM, Erik Froese <span dir="ltr"><<a href="mailto:erik.froese@gmail.com" target="_blank">erik.froese@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Is each OST journals on its own physical disk? I've seen those messages when there isn't enough hardware dedicated to the journal device.<div>Erik<div><div></div><div><div><br><div class="gmail_quote">On Sun, Jan 24, 2010 at 11:43 PM, Aaron Knister <span dir="ltr"><<a href="mailto:aaron.knister@gmail.com" target="_blank">aaron.knister@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;"><div>I don't necessarily think there's anything wrong with using drbd or running it over gigabit ethernet. If you stop all I/O to the lustre filesystem, what does an hdparm -t show on the sdc and drbd devices? Do you have any performance numbers for the drbd or underlying raid devices?</div>
<div><div></div><div><br><div><div>On Jan 24, 2010, at 11:17 PM, Lex wrote:</div><br><blockquote type="cite">Thank you for your fast reply, Aaron<br><br>I'm using Giga Ethernet to synchronize data between to our fail-over node. Is there something wrong ? Tell me, please <br>
<br><div class="gmail_quote">On Mon, Jan 25, 2010 at 10:35 AM, Aaron Knister <span dir="ltr"><<a href="mailto:aaron.knister@gmail.com" target="_blank">aaron.knister@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><div style="word-wrap: break-word;">My best guess (and please correct me if I'm wrong) is that those messages are because the underlying block devices are slow to respond to i/o requests. It looks like you're using DRBD. What's your interconnect? <div>
<br><div><div><div></div><div><div>On Jan 24, 2010, at 9:42 PM, Lex wrote:</div><br></div></div><blockquote type="cite"><div><div></div><div>Hi list <br><br>I have one OSS with hadware info like this : <br>
<br>CPU Intel(R) xeon E5420 2.5 Ghz<br>Chipset intel 5000P <br>8GB RAM <br><br>With this OSS, we using 2 RAID-5 arrays as OSTs ( each has 4 x 1.5 TB hard drive with RAID controller adaptec 5805 ) <br>
<br>I worked quite smooth before, but, about 2 weeks ago, in /var/log/messages, i saw many warning ( i thought so) like this: <br><br><i>Jan 25 08:41:23 OST6 kernel: Lustre: 9587:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 35s<br>
Jan 25 08:41:34 OST6 kernel: Lustre: 9608:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 41s<br>Jan 25 08:41:34 OST6 kernel: Lustre: 9608:0:(filter_io_26.c:706:filter_commitrw_write()) Skipped 2 previous similar messages<br>
Jan 25 08:41:35 OST6 kernel: Lustre: 9645:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 43s<br>Jan 25 08:58:10 OST6 kernel: Lustre: 9646:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 31s<br>
Jan 25 08:59:39 OST6 kernel: Lustre: 9609:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 30s<br>Jan 25 09:01:05 OST6 kernel: Lustre: 9587:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 33s<br>
Jan 25 09:03:23 OST6 kernel: Lustre: 9633:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 32s<br>Jan 25 09:11:25 OST6 kernel: Lustre: 9585:0:(filter_io_26.c:706:filter_commitrw_write()) lustre-OST0006: slow direct_io 36s</i><br>
<br>I googled around and found that it's because a problem with oss_num_threads and even though brought it down to 64 ( followed by the function i found in the 1.8 manual: thread_number = RAM * CPU core / 128 MB, its value is 256 ) <br>
<br><i>options ost oss_num_threads=64</i><br><br>It still didn't help. <br><br>I thought it was only the harmless warning but maybe wrong, our performance is goes down quite heavily ( it's maybe because of other reason, but for now, i am only doubting slow direct_io problem ) <br>
<br>iostat -m 1 1<br>Linux 2.6.18-92.1.17.el5_lustre.1.8.0custom (OST6) 01/25/2010<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.01 0.02 2.86 25.01 0.00 72.10<br><br>Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn<br>
sda 1.30 0.01 0.00 11386 3469<br>sdb 1.30 0.01 0.00 11531 3469<br>sdc 131.50 <b>12.40</b> 0.26 11793218 249934<br>
sdd 178.46 <b>18.00</b> 0.26 17124065 250334<br>md2 3.33 0.02 0.00 22915 2634<br>md1 0.00 0.00 0.00 0 0<br>
md0 0.00 0.00 0.00 0 0<br>drbd3 480.10 <b>12.39</b> 0.26 11789047 249639<br>drbd6 565.85 <b>14.89</b> 0.26 14168452 249211<br>
<br><br>So, could anyone please tell me whether it's warning impact our system performance or not ? and if it does, give me solution or advice to resolve it, please <br><br>Best regards <br><br><br><br><br><br><br></div>
</div>
_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
</blockquote></div><br></div></div></blockquote></div><br>
</blockquote></div><br></div></div></div><br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br></div></div></div></div>
</blockquote></div><br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br>
</div></div></blockquote></div><br>