<br><div class="gmail_quote">From: <b class="gmail_sendername">Lex</b> <span dir="ltr"><<a href="mailto:lexluthor87@gmail.com">lexluthor87@gmail.com</a>></span><br>Date: Mon, Feb 1, 2010 at 10:28 PM<br>Subject: Re: [Lustre-discuss] High difference in I/O network traffic in lustre client<br>
To: Mag Gam <<a href="mailto:magawake@gmail.com">magawake@gmail.com</a>><br><br><br>I have 8 OSSs and 8 OSTs. Hadware info: <br><br>CPU Intel(R) xeon E5420 2.5 Ghz Chipset intel 5000P<br>8GB RAM<br>8 x 1.5TB hard disks, divided into 2 arrays with raid controller adaptec 5805<br>
<br>We using 2 x 1Gigabit Ethernet card with linux bonding ( OS is centos 5.3 ). Our lustre client work as web server for downloading file, so there are many files has been read by web client, i can't provide you an exact number. ( we have about millions file in our lustre storage system, unfortunately, there are quite a lot small file: a linux soft links ) Files are "striped" over each 2 OSTs, some are striped over all our OSTs ( fewer than 2 OSTs parallel striping ) <br>
<br>Do you have any idea for my issue ? <br><br>Many thanks <br><div><div></div><div class="h5"><br><br><div class="gmail_quote">On Mon, Feb 1, 2010 at 8:05 PM, Mag Gam <span dir="ltr"><<a href="mailto:magawake@gmail.com" target="_blank">magawake@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">How many OSS and OSTs do you have ? What type of hardware are they<br>
running on? What type of network connection? The file you are trying<br>
to access what OSS is it on? Are the files striped?<br>
<br>
<br>
<br>
What<br>
<div><div></div><div><br>
On Mon, Feb 1, 2010 at 4:44 AM, Lex <<a href="mailto:lexluthor87@gmail.com" target="_blank">lexluthor87@gmail.com</a>> wrote:<br>
> Hi guys<br>
><br>
> In effort to improve our storage system performance, i found some strange<br>
> signs but unfortunately, couldn't explain it by myself. So i post here for<br>
> all you guys can't help me to clarify it<br>
><br>
> I'm using lustre client as web server for downloading file. When our system<br>
> in a heavy load ( about 12.000 concurrent connection for 8 web server -<br>
> lustre client ), %iowait has been pushed to about 98%, load average was<br>
> about 1-2000 !!!! ( just because of %iowait, i still could manipulate<br>
> normally almost every command over ssh ) i think it's a terrible number in<br>
> describing load average ! But, at that case, the in and out network traffic<br>
> are almost the same ( although just about few MB/s :( )<br>
><br>
> The odd thing is, right now, when we only have about 3.500 concurrent<br>
> connection, load average is about 50 ( still too big, right ? ), iowait is<br>
> about 70%, the difference between receive and transmit network is too hight,<br>
> about 10-20MB ( see attached file, please )<br>
><br>
> We just have about 20 connection for our local lustre storage system:<br>
><br>
> netstat -nat | grep 192.168.1.75<br>
> tcp 0 560 <a href="http://192.168.1.75:1023" target="_blank">192.168.1.75:1023</a> <a href="http://192.168.1.85:988" target="_blank">192.168.1.85:988</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:1023" target="_blank">192.168.1.75:1023</a> <a href="http://192.168.1.81:988" target="_blank">192.168.1.81:988</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:988" target="_blank">192.168.1.75:988</a> <a href="http://192.168.1.85:1023" target="_blank">192.168.1.85:1023</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:988" target="_blank">192.168.1.75:988</a> <a href="http://192.168.1.85:1022" target="_blank">192.168.1.85:1022</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:988" target="_blank">192.168.1.75:988</a> <a href="http://192.168.1.81:1023" target="_blank">192.168.1.81:1023</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:988" target="_blank">192.168.1.75:988</a> <a href="http://192.168.1.81:1022" target="_blank">192.168.1.81:1022</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:988" target="_blank">192.168.1.75:988</a> <a href="http://192.168.1.100:1023" target="_blank">192.168.1.100:1023</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:1021" target="_blank">192.168.1.75:1021</a> <a href="http://192.168.1.78:988" target="_blank">192.168.1.78:988</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:1023" target="_blank">192.168.1.75:1023</a> <a href="http://192.168.1.78:988" target="_blank">192.168.1.78:988</a><br>
> ESTABLISHED<br>
> tcp 0 0 <a href="http://192.168.1.75:1022" target="_blank">192.168.1.75:1022</a> <a href="http://192.168.1.78:988" target="_blank">192.168.1.78:988</a><br>
> ESTABLISHED<br>
> tcp 0 560 <a href="http://192.168.1.75:1023" target="_blank">192.168.1.75:1023</a> <a href="http://192.168.1.100:988" target="_blank">192.168.1.100:988</a><br>
> ESTABLISHED<br>
><br>
> and about 400 connection with client from internet :<br>
><br>
> netstat -nat | grep out_wan_ip | grep EST | wc -l<br>
> 407<br>
><br>
> We're currently using 2 Gigabit Ethernet card, one for <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a><br>
> network for lnet and the other as wan ip for delivering file out to internet<br>
> and about 15MB/s thoughput was "lost" somehow !!!!<br>
><br>
> So, my question is:<br>
><br>
> - Is there anyone have idea or hint about high load situation with our<br>
> lustre client - web server like i described above ? I followed this link<br>
> and found out kjournald process is the main main "culprit" ( with our ost,<br>
> it was "ll" process )<br>
> - What makes the too high difference between receive and transit direction<br>
> in our lustre client - web server ?<br>
><br>
><br>
> i'm really stressed with poor performance in our storage system and hope<br>
> anyone here can help me point out some thing<br>
><br>
> Any help would be highly appreciated<br>
><br>
> Best regards<br>
><br>
><br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Lustre-discuss mailing list<br>
> <a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
><br>
><br>
</blockquote></div><br>
</div></div></div><br>