Are you sure the IB card is connected at QDR rate?<div><br></div><div>ibstat will tell you. Look at the "Rate" line.</div><div><br></div><div>What kind of machines are you using?</div><div><br></div><div><div>ibstat </div>
<div>CA 'mlx4_0'</div><div> CA type: MT26428</div><div> Number of ports: 2</div><div> Firmware version: 2.6.0</div><div> Hardware version: a0</div><div> Node GUID: 0x00212800013e5432</div>
<div> System image GUID: 0x00212800013e5435</div><div> Port 1:</div><div> State: Active</div><div> Physical state: LinkUp</div><div> Rate: 40</div><div> Base lid: 13</div>
<div> LMC: 0</div><div> SM lid: 2</div><div> Capability mask: 0x02510868</div><div> Port GUID: 0x00212800013e5433</div><div> Port 2:</div><div> State: Active</div>
<div> Physical state: LinkUp</div><div> Rate: 40</div><div> Base lid: 14</div><div> LMC: 0</div><div> SM lid: 2</div><div> Capability mask: 0x02510868</div>
<div> Port GUID: 0x00212800013e5434</div><div><br></div><br><div class="gmail_quote">On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy <span dir="ltr"><<a href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>
ThanQ Atul<br>
<br>
I have done the following test, According to the below test, it reaching 1869.74 MB/sec.<br>
<br>
This is the command I used for benchmarking.<br>
<br>
# mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v -o /mnt/file1<br>
<br>
Max Write: 421.31 MiB/sec (441.77 MB/sec)<br>Max Read: 1029.68 MiB/sec (1079.70 MB/sec)<br><br>***<br>
1. In the aboove command I used MPIIO with 1024 process, I got the almost same result when I fired same command with 512 process<br>
<br>
2. I tested with stripe and without stripe with both the apis MPIIO and POSIX<br>
<br>
<br>
**<br>
node0:~ # ib_write_bw<br>------------------------------------------------------------------<br> RDMA_Write BW Test<br>Number of qp's running 1<br>Connection type : RC<br>Each Qp will post up to 100 messages each time<br>
Inline data is used up to 1 bytes message<br> local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr 0x002b5fd9964000<br> remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr 0x002ae797667000<br>
Mtu : 2048<br>------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<br>node0:~ #<br>
node1:~ # ib_write_bw node0<br>------------------------------------------------------------------<br> RDMA_Write BW Test<br>Number of qp's running 1<br>Connection type : RC<br>Each Qp will post up to 100 messages each time<br>
Inline data is used up to 1 bytes message<br> local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr 0x002ae797667000<br> remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr 0x002b5fd9964000<br>
Mtu : 2048<br>------------------------------------------------------------------<br> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<br> 65536 5000 1869.76 1869.74<br>------------------------------------------------------------------<br>
node1:~ #<br><br><br>
<br> <br>
<hr>
Date: Thu, 31 Dec 2009 17:43:44 +1100<br>From: Atul.Vidwansa@Sun.COM<br>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<br>To: <a href="mailto:klakshman03@hotmail.com" target="_blank">klakshman03@hotmail.com</a><br>
CC: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><div><div></div><div class="h5"><br><br>
<div>Hi Lakshmana,</div>
<div><br></div>
<div>You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth between pair of nodes.</div>
<div><br></div>
<div>Lustre provides lnet selftest tool for measuring IB bandwidth between pair or group of nodes. Check lustre manual for how tos.</div>
<div><br></div>
<div>Finally, can you post IOR parameters you are using?</div>
<div><br></div>
<div>Cheers,</div>
<div>Atul</div>
<div><br></div>
<div><br><br>Sent from my iPhone</div>
<div><br>On Dec 31, 2009, at 5:13 PM, lakshmana swamy <<a href="mailto:klakshman03@hotmail.com" target="_blank">klakshman03@hotmail.com</a>> wrote:<br><br></div>
<div></div>
<blockquote>
<div> <br> Dear All,<br> <br> Whats the maximum write speed can be achieved with Infiniband QDR HCA card.<br> <br> <br> The Question arised from my lustre setup where Iam not able to exceed 500MB/s (write speed) with two OSSs and one MDS<br>
<br> Each OSS has four OSTs and one HCA card. I would like to know which is the bottleneck here.<br> <br> <br> I have been using IOR for benchmarking<br> <br>Thank you<br> <br>LakshmaN<br><br>
<hr>
New Windows 7: Simplify what you do everyday. <a href="http://windows.microsoft.com/shop" target="_blank">Find the right PC for you.</a> </div></blockquote>
<blockquote>
<div><span>_______________________________________________</span><br><span>Lustre-discuss mailing list</span><br><span><a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a></span><br>
<span><a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a></span><br></div></blockquote> <br></div></div><hr>New Windows 7: Find the right PC for you. <a href="http://windows.microsoft.com/shop" target="_blank">Learn more.</a></div>
<br>_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br></blockquote></div><br></div>