<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Appreciate the input.<br><div><br></div><div>We've been using mode 6 as I expect it provides the fewest configuration pratfalls. IF the single stream becomes our bottleneck we'll mess with aggregation.</div><div><br></div><div>What I can't find is the bottleneck in our current setup. With 4 servers - 2 clients, two OSSs - I'd expect 4Gb of aggregate throughput where each client has a single connection to each OST. Instead we're limited to 2GB, where each OSS appears limited to 1Gb of I/O. The strangeness is that iptraf on the OSSs shows traffic through the expected connections (2 X 2) but at only 35% - 65% of bandwidth. </div><div><br></div><div>And a third client writing to the filesystem will briefly increase aggregate throughput, but it quickly settles back to ~2Gb.</div><div><br></div><div>djm</div><div><br></div><div><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px;"><span class="Apple-style-span" style="font-size: medium;"><br></span></span></font></div><div><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><br class="Apple-interchange-newline"></span></div></span> </div><br><div><div>On Jan 27, 2011, at 11:16 AM, Kevin Van Maren wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Normally if you are having a problem with write BW, you need to futz with the switch. If you were having<br>problems with read BW, you need to futz with the server's config (xmit hash policy is the usual culprit).<br><br>Are you testing multiple clients to the same server?<br><br>Are you using mode 6 because you don't have bonding support in your switch? I normally use 802.3ad mode,<br>assuming your switch supports link aggregation.<br><br><br>I was bonding 2x1Gb links for Lustre back in 2004. That was before BOND_XMIT_POLICY_LAYER34<br>was in the kernel, so I had to hack the bond xmit hash (with multiple NICs standard, layer2 hashing does not<br>produce a uniform distribution, and can't work if going through a router).<br><br>Any one connection (socket or node/node connection) will use only one gigabit link. While it is possible<br>to use two links using round-robin, that normally only helps for client reads (server can't choose which link to<br>receive data, the switch picks that), and has the serious downside of out-of-order packets on the TCP stream.<br><br>[If you want clients to have better client bandwidth for a single file, change your default stripe count to 2, so it<br>will hit two different servers.]<br><br>Kevin<br><br><br>David Merhar wrote:<br><blockquote type="cite">Sorry - little b all the way around.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We're limited to 1Gb per OST.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">djm<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Jan 27, 2011, at 7:48 AM, Balagopal Pillai wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> <br></blockquote><blockquote type="cite"><blockquote type="cite">I guess you have two gigabit nics bonded in mode 6 and not two 1GB nics?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">(B-Bytes, b-bits) The max aggregate throughput could be about 200MBps<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">out of the 2 bonded nics. I think the mode 0 bonding works only with<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">cisco etherchannel or something similar on the switch side. Same with<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the FC connection, its 4Gbps (not 4GBps) or about 400-500 MBps max<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">throughout. Maybe you could also see the max read and write capabilities<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">of the raid controller other than just the network. When testing with<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">dd, some of the data remains as dirty data till its flushed into the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">disk. I think the default background ratio is 10% for rhel5 which would<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">be sizable if your oss have lots of ram. There is chance of lockup of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the oss once it hits the dirty_ratio limit,which is 40% by default. So a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bit more aggressive flush to disk by lowering the background_ratio and a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bit more headroom before it hits the dirty_ratio is generally desirable<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">if your raid controller could keep up with it. So with your current<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">setup, i guess you could get a max of 400MBps out of both OSS's if they<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">both have two 1Gb nics in them. Maybe if you have one of the switches<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">from Dell that has 4 10Gb ports in them (their powerconnect 6248), 10Gb<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">nics for your OSS's might be a cheaper way to increase the aggregate<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">performance. I think over 1GBps from a client is possible in cases where<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">you use infiniband and rdma to deliver data.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">David Merhar wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Our OSS's with 2x1GB NICs (bonded) appear limited to 1GB worth of<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">write throughput each.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Our setup:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">2 OSS serving 1 OST each<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Lustre 1.8.5<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">RHEL 5.4<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">New Dell M610's blade servers with plenty of CPU and RAM<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">All SAN fibre connections are at least 4GB<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Some notes:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- A direct write (dd) from a single OSS to the OST gets 4GB, the OSS's<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">fibre wire speed.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- A single client will get 2GB of lustre write speed, the client's<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">ethernet wire speed.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- We've tried bond mode 6 and 0 on all systems. With mode 6 we will<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">see both NICs on both OSSs receiving data.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">- We've tried multiple OSTs per OSS.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">But 2 clients writing a file will get 2GB of total bandwidth to the<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">filesystems. We have been unable to isolate any particular resource<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">bottleneck. None of the systems (MDS, OSS, or client) seem to be<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">working very hard.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The 1GB per OSS threshold is so consistent, that it almost appears by<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">design - and hopefully we're missing something obvious.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Any advice?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Thanks.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">djm<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Lustre-discuss mailing list<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"> <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Lustre-discuss mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> <br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Lustre-discuss mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br></blockquote><blockquote type="cite"><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></blockquote><blockquote type="cite"> <br></blockquote></div></blockquote></div><br></div></body></html>