<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Iam Sorry,<BR>
<BR>
It is DDR not QDR...but still which is the bottleneck<BR>
<BR>
<BR>
Thank you<BR>
<BR>
laxman <BR> <BR>
<HR id=stopSpelling>
Date: Thu, 31 Dec 2009 12:12:40 -0500<BR>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<BR>From: erik.froese@gmail.com<BR>To: klakshman03@hotmail.com<BR>CC: atul.vidwansa@sun.com; lustre-discuss@lists.lustre.org<BR><BR>Are you sure the IB card is connected at QDR rate?
<DIV><BR></DIV>
<DIV>ibstat will tell you. Look at the "Rate" line.</DIV>
<DIV><BR></DIV>
<DIV>What kind of machines are you using?</DIV>
<DIV><BR></DIV>
<DIV>
<DIV>ibstat </DIV>
<DIV>CA 'mlx4_0'</DIV>
<DIV> CA type: MT26428</DIV>
<DIV> Number of ports: 2</DIV>
<DIV> Firmware version: 2.6.0</DIV>
<DIV> Hardware version: a0</DIV>
<DIV> Node GUID: 0x00212800013e5432</DIV>
<DIV> System image GUID: 0x00212800013e5435</DIV>
<DIV> Port 1:</DIV>
<DIV> State: Active</DIV>
<DIV> Physical state: LinkUp</DIV>
<DIV> Rate: 40</DIV>
<DIV> Base lid: 13</DIV>
<DIV> LMC: 0</DIV>
<DIV> SM lid: 2</DIV>
<DIV> Capability mask: 0x02510868</DIV>
<DIV> Port GUID: 0x00212800013e5433</DIV>
<DIV> Port 2:</DIV>
<DIV> State: Active</DIV>
<DIV> Physical state: LinkUp</DIV>
<DIV> Rate: 40</DIV>
<DIV> Base lid: 14</DIV>
<DIV> LMC: 0</DIV>
<DIV> SM lid: 2</DIV>
<DIV> Capability mask: 0x02510868</DIV>
<DIV> Port GUID: 0x00212800013e5434</DIV>
<DIV><BR></DIV><BR>
<DIV class=ecxgmail_quote>On Thu, Dec 31, 2009 at 2:04 AM, lakshmana swamy <SPAN dir=ltr><<A href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE style="BORDER-LEFT: #ccc 1px solid; PADDING-LEFT: 1ex" class=ecxgmail_quote>
<DIV>ThanQ Atul<BR> <BR>I have done the following test, According to the below test, it reaching 1869.74 MB/sec.<BR> <BR>This is the command I used for benchmarking.<BR> <BR># mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v -o /mnt/file1<BR> <BR>Max Write: 421.31 MiB/sec (441.77 MB/sec)<BR>Max Read: 1029.68 MiB/sec (1079.70 MB/sec)<BR><BR>***<BR>1. In the aboove command I used MPIIO with 1024 process, I got the almost same result when I fired same command with 512 process<BR> <BR>2. I tested with stripe and without stripe with both the apis MPIIO and POSIX<BR> <BR> <BR>**<BR>node0:~ # ib_write_bw<BR>------------------------------------------------------------------<BR> RDMA_Write BW Test<BR>Number of qp's running 1<BR>Connection type : RC<BR>Each Qp will post up to 100 messages each time<BR>Inline data is used up to 1 bytes message<BR> local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr 0x002b5fd9964000<BR> remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr 0x002ae797667000<BR>Mtu : 2048<BR>------------------------------------------------------------------<BR> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<BR>node0:~ #<BR>node1:~ # ib_write_bw node0<BR>------------------------------------------------------------------<BR> RDMA_Write BW Test<BR>Number of qp's running 1<BR>Connection type : RC<BR>Each Qp will post up to 100 messages each time<BR>Inline data is used up to 1 bytes message<BR> local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr 0x002ae797667000<BR> remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr 0x002b5fd9964000<BR>Mtu : 2048<BR>------------------------------------------------------------------<BR> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<BR> 65536 5000 1869.76 1869.74<BR>------------------------------------------------------------------<BR>node1:~ #<BR><BR><BR><BR> <BR>
<HR>
Date: Thu, 31 Dec 2009 17:43:44 +1100<BR>From: Atul.Vidwansa@Sun.COM<BR>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<BR>To: <A href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</A><BR>CC: <A href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</A>
<DIV>
<DIV></DIV>
<DIV class=h5><BR><BR>
<DIV>Hi Lakshmana,</DIV>
<DIV><BR></DIV>
<DIV>You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth between pair of nodes.</DIV>
<DIV><BR></DIV>
<DIV>Lustre provides lnet selftest tool for measuring IB bandwidth between pair or group of nodes. Check lustre manual for how tos.</DIV>
<DIV><BR></DIV>
<DIV>Finally, can you post IOR parameters you are using?</DIV>
<DIV><BR></DIV>
<DIV>Cheers,</DIV>
<DIV>Atul</DIV>
<DIV><BR></DIV>
<DIV><BR><BR>Sent from my iPhone</DIV>
<DIV><BR>On Dec 31, 2009, at 5:13 PM, lakshmana swamy <<A href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</A>> wrote:<BR><BR></DIV>
<DIV></DIV>
<BLOCKQUOTE>
<DIV> <BR> Dear All,<BR> <BR> Whats the maximum write speed can be achieved with Infiniband QDR HCA card.<BR> <BR> <BR> The Question arised from my lustre setup where Iam not able to exceed 500MB/s (write speed) with two OSSs and one MDS<BR> <BR> Each OSS has four OSTs and one HCA card. I would like to know which is the bottleneck here.<BR> <BR> <BR> I have been using IOR for benchmarking<BR> <BR>Thank you<BR> <BR>LakshmaN<BR><BR>
<HR>
New Windows 7: Simplify what you do everyday. <A href="http://windows.microsoft.com/shop">Find the right PC for you.</A> </DIV></BLOCKQUOTE>
<BLOCKQUOTE>
<DIV><SPAN>_______________________________________________</SPAN><BR><SPAN>Lustre-discuss mailing list</SPAN><BR><SPAN><A href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</A></SPAN><BR><SPAN><A href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</A></SPAN><BR></DIV></BLOCKQUOTE><BR></DIV></DIV>
<HR>
New Windows 7: Find the right PC for you. <A href="http://windows.microsoft.com/shop">Learn more.</A></DIV><BR>_______________________________________________<BR>Lustre-discuss mailing list<BR><A href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</A><BR><A href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</A><BR><BR></BLOCKQUOTE></DIV><BR></DIV> <br /><hr />http://windows.microsoft.com/shop <a href='Windows 7: Simplify what you do everyday.' target='_new'>Find the right PC for you.</a></body>
</html>