Hi guys<br><br>In effort to improve our storage system performance, i found some strange signs but unfortunately, couldn't explain it by myself. So i post here for all you guys can't help me to clarify it<br><br>I'm using lustre client as web server for downloading file. When our system in a heavy load ( about 12.000 concurrent connection for 8 web server - lustre client ), %iowait has been pushed to about 98%, load average was about 1-2000 !!!! ( just because of %iowait, i still could manipulate normally almost every command over ssh ) i think it's a terrible number in describing load average ! But, at that case, the in and out network traffic<b> are almost the same</b> ( although just about few MB/s :( ) <br>
<br>The odd thing is, right now, when we only have about 3.500 concurrent connection, load average is about 50 ( still too big, right ? ), iowait is about 70%, the difference between receive and transmit network is too hight, about 10-20MB ( see attached file, please ) <br>
<br>We just have about 20 connection for our local lustre storage system: <br><br><i>netstat -nat | grep 192.168.1.75<br>tcp 0 560 <a href="http://192.168.1.75:1023">192.168.1.75:1023</a> <a href="http://192.168.1.85:988">192.168.1.85:988</a> ESTABLISHED<br>
tcp 0 0 <a href="http://192.168.1.75:1023">192.168.1.75:1023</a> <a href="http://192.168.1.81:988">192.168.1.81:988</a> ESTABLISHED<br>tcp 0 0 <a href="http://192.168.1.75:988">192.168.1.75:988</a> <a href="http://192.168.1.85:1023">192.168.1.85:1023</a> ESTABLISHED<br>
tcp 0 0 <a href="http://192.168.1.75:988">192.168.1.75:988</a> <a href="http://192.168.1.85:1022">192.168.1.85:1022</a> ESTABLISHED<br>tcp 0 0 <a href="http://192.168.1.75:988">192.168.1.75:988</a> <a href="http://192.168.1.81:1023">192.168.1.81:1023</a> ESTABLISHED<br>
tcp 0 0 <a href="http://192.168.1.75:988">192.168.1.75:988</a> <a href="http://192.168.1.81:1022">192.168.1.81:1022</a> ESTABLISHED<br>tcp 0 0 <a href="http://192.168.1.75:988">192.168.1.75:988</a> <a href="http://192.168.1.100:1023">192.168.1.100:1023</a> ESTABLISHED<br>
tcp 0 0 <a href="http://192.168.1.75:1021">192.168.1.75:1021</a> <a href="http://192.168.1.78:988">192.168.1.78:988</a> ESTABLISHED<br>tcp 0 0 <a href="http://192.168.1.75:1023">192.168.1.75:1023</a> <a href="http://192.168.1.78:988">192.168.1.78:988</a> ESTABLISHED<br>
tcp 0 0 <a href="http://192.168.1.75:1022">192.168.1.75:1022</a> <a href="http://192.168.1.78:988">192.168.1.78:988</a> ESTABLISHED<br>tcp 0 560 <a href="http://192.168.1.75:1023">192.168.1.75:1023</a> <a href="http://192.168.1.100:988">192.168.1.100:988</a> ESTABLISHED</i><br>
<br>and about 400 connection with client from internet : <br><br><i>netstat -nat | grep out_wan_ip | grep EST | wc -l<br>407</i><br><br>We're currently using 2 Gigabit Ethernet card, one for <a href="http://192.168.1.0/24">192.168.1.0/24</a> network for lnet and the other as wan ip for delivering file out to internet and <b>about 15MB/s thoughput was "lost" somehow</b> !!!! <br>
<br>So, my question is: <br><br>- Is there anyone have idea or hint about high load situation with our lustre client - web server like i described above ? I followed this <a href="http://rackerhacker.com/2008/03/11/hunting-down-elusive-sources-of-iowait/">link </a>and found out <i><b>kjournald </b></i>process is the main main "culprit" ( with our ost, it was "<b>ll</b>" process ) <br>
- What makes the too high difference between receive and transit direction in our lustre client - web server ? <br><br><br>i'm really stressed with poor performance in our storage system and hope anyone here can help me point out some thing <br>
<br>Any help would be highly appreciated <br><br>Best regards <br><br><br><i><b></b><br></i><br>