<html><head><base href="x-msg://14/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>If you do indeed have DDR infiniband, then your ib_write_bw tests are achieving fairly close to the maximum possible throughput of 2gigabytes/second. As far as where your bottleneck lies, my guess is within the storage connected to your OSSs. What are you using for back-end storage?</div><br><div><div>On Jan 1, 2010, at 1:02 AM, lakshmana swamy wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="hmmessage" style="font-size: 10pt; font-family: Verdana; ">Iam Sorry,<br> <br>It is DDR not QDR...but still which is the bottleneck<br> <br> <br>Thank you<br> <br>laxman <br> <br><hr id="stopSpelling">Date: Thu, 31 Dec 2009 12:12:40 -0500<br>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<br>From:<span class="Apple-converted-space"> </span><a href="mailto:erik.froese@gmail.com">erik.froese@gmail.com</a><br>To:<span class="Apple-converted-space"> </span><a href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</a><br>CC:<span class="Apple-converted-space"> </span><a href="mailto:atul.vidwansa@sun.com">atul.vidwansa@sun.com</a>;<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><br><br>Are you sure the IB card is connected at QDR rate?<div><br></div><div>ibstat will tell you. Look at the "Rate" line.</div><div><br></div><div>What kind of machines are you using?</div><div><br></div><div><div>ibstat </div><div>CA 'mlx4_0'</div><div> CA type: MT26428</div><div> Number of ports: 2</div><div> Firmware version: 2.6.0</div><div> Hardware version: a0</div><div> Node GUID: 0x00212800013e5432</div><div> System image GUID: 0x00212800013e5435</div><div> Port 1:</div><div> State: Active</div><div> Physical state: LinkUp</div><div> Rate: 40</div><div> Base lid: 13</div><div> LMC: 0</div><div> SM lid: 2</div><div> Capability mask: 0x02510868</div><div> Port GUID: 0x00212800013e5433</div><div> Port 2:</div><div> State: Active</div><div> Physical state: LinkUp</div><div> Rate: 40</div><div> Base lid: 14</div><div> LMC: 0</div><div> SM lid: 2</div><div> Capability mask: 0x02510868</div><div> Port GUID: 0x00212800013e5434</div><div><br></div><br><div class="ecxgmail_quote">On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="ecxgmail_quote" style="border-left-color: rgb(204, 204, 204); border-left-width: 1px; border-left-style: solid; padding-left: 1ex; "><div>ThanQ Atul<br> <br>I have done the following test, According to the below test, it reaching 1869.74 MB/sec.<br> <br>This is the command I used for benchmarking.<br> <br># mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v -o /mnt/file1<br> <br>Max Write: 421.31 MiB/sec (441.77 MB/sec)<br>Max Read: 1029.68 MiB/sec (1079.70 MB/sec)<br><br>***<br>1. In the aboove command I used MPIIO with 1024 process, I got the almost same result when I fired same command with 512 process<br> <br>2. I tested with stripe and without stripe with both the apis MPIIO and POSIX<br> <br> <br>**<br>node0:~ # ib_write_bw<br>------------------------------------------------------------------<br> RDMA_Write BW Test<br>Number of qp's running 1<br>Connection type : RC<br>Each Qp will post up to 100 messages each time<br>Inline data is used up to 1 bytes message<br> local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr 0x002b5fd9964000<br> remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr 0x002ae797667000<br>Mtu : 2048<br>------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<br>node0:~ #<br>node1:~ # ib_write_bw node0<br>------------------------------------------------------------------<br> RDMA_Write BW Test<br>Number of qp's running 1<br>Connection type : RC<br>Each Qp will post up to 100 messages each time<br>Inline data is used up to 1 bytes message<br> local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr 0x002ae797667000<br> remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr 0x002b5fd9964000<br>Mtu : 2048<br>------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<br> 65536 5000 1869.76 1869.74<br>------------------------------------------------------------------<br>node1:~ #<br><br><br><br> <br><hr>Date: Thu, 31 Dec 2009 17:43:44 +1100<br>From:<span class="Apple-converted-space"> </span><a href="mailto:Atul.Vidwansa@Sun.COM">Atul.Vidwansa@Sun.COM</a><br>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<br>To:<span class="Apple-converted-space"> </span><a href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</a><br>CC:<span class="Apple-converted-space"> </span><a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a><div><div></div><div class="h5"><br><br><div>Hi Lakshmana,</div><div><br></div><div>You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth between pair of nodes.</div><div><br></div><div>Lustre provides lnet selftest tool for measuring IB bandwidth between pair or group of nodes. Check lustre manual for how tos.</div><div><br></div><div>Finally, can you post IOR parameters you are using?</div><div><br></div><div>Cheers,</div><div>Atul</div><div><br></div><div><br><br>Sent from my iPhone</div><div><br>On Dec 31, 2009, at 5:13 PM, lakshmana swamy <<a href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</a>> wrote:<br><br></div><div></div><blockquote><div> <br> Dear All,<br> <br> Whats the maximum write speed can be achieved with Infiniband QDR HCA card.<br> <br> <br> The Question arised from my lustre setup where Iam not able to exceed 500MB/s (write speed) with two OSSs and one MDS<br> <br> Each OSS has four OSTs and one HCA card. I would like to know which is the bottleneck here.<br> <br> <br> I have been using IOR for benchmarking<br> <br>Thank you<br> <br>LakshmaN<br><br><hr>New Windows 7: Simplify what you do everyday.<span class="Apple-converted-space"> </span><a href="http://windows.microsoft.com/shop">Find the right PC for you.</a></div></blockquote><blockquote><div><span>_______________________________________________</span><br><span>Lustre-discuss mailing list</span><br><span><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a></span><br><span><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a></span><br></div></blockquote><br></div></div><hr>New Windows 7: Find the right PC for you.<span class="Apple-converted-space"> </span><a href="http://windows.microsoft.com/shop">Learn more.</a></div><br>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br><br></blockquote></div><br></div><br><hr><a href="http://windows.microsoft.com/shop">http://windows.microsoft.com/shop</a><span class="Apple-converted-space"> </span><a href="Windows 7: Simplify what you do everyday." target="_new">Find the right PC for you.</a><span class="Apple-converted-space"> </span>_______________________________________________<br>Lustre-discuss mailing list<br><a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br></div></span></blockquote></div><br></body></html>