<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
ThanQ Atul<BR>
<BR>
I have done the following test, According to the below test, it reaching 1869.74 MB/sec.<BR>
<BR>
This is the command I used for benchmarking.<BR>
<BR>
# mpiexec -machinefile machs1 -np 1024 IOR -N 1024 -a MPIIO -t 1024K -b 1G -F -v -o /mnt/file1<BR>
<BR>
Max Write: 421.31 MiB/sec (441.77 MB/sec)<BR>Max Read: 1029.68 MiB/sec (1079.70 MB/sec)<BR><BR>***<BR>
1. In the aboove command I used MPIIO with 1024 process, I got the almost same result when I fired same command with 512 process<BR>
<BR>
2. I tested with stripe and without stripe with both the apis MPIIO and POSIX<BR>
<BR>
<BR>
**<BR>
node0:~ # ib_write_bw<BR>------------------------------------------------------------------<BR> RDMA_Write BW Test<BR>Number of qp's running 1<BR>Connection type : RC<BR>Each Qp will post up to 100 messages each time<BR>Inline data is used up to 1 bytes message<BR> local address: LID 0x100, QPN 0x28004e, PSN 0xc17c83 RKey 0x8002d00 VAddr 0x002b5fd9964000<BR> remote address: LID 0x101, QPN 0x240051, PSN 0x5e61e, RKey 0x8002d00 VAddr 0x002ae797667000<BR>Mtu : 2048<BR>------------------------------------------------------------------<BR> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<BR>node0:~ #<BR>
node1:~ # ib_write_bw node0<BR>------------------------------------------------------------------<BR> RDMA_Write BW Test<BR>Number of qp's running 1<BR>Connection type : RC<BR>Each Qp will post up to 100 messages each time<BR>Inline data is used up to 1 bytes message<BR> local address: LID 0x101, QPN 0x240051, PSN 0x5e61e RKey 0x8002d00 VAddr 0x002ae797667000<BR> remote address: LID 0x100, QPN 0x28004e, PSN 0xc17c83, RKey 0x8002d00 VAddr 0x002b5fd9964000<BR>Mtu : 2048<BR>------------------------------------------------------------------<BR> #bytes #iterations BW peak[MB/sec] BW average[MB/sec]<BR> 65536 5000 1869.76 1869.74<BR>------------------------------------------------------------------<BR>node1:~ #<BR><BR><BR>
<BR> <BR>
<HR id=stopSpelling>
Date: Thu, 31 Dec 2009 17:43:44 +1100<BR>From: Atul.Vidwansa@Sun.COM<BR>Subject: Re: [Lustre-discuss] max write speed of QDR HCA card<BR>To: klakshman03@hotmail.com<BR>CC: lustre-discuss@lists.lustre.org<BR><BR>
<DIV>Hi Lakshmana,</DIV>
<DIV><BR></DIV>
<DIV>You can use standard ofed tools like ib_rdma_bw to check Infiniband bandwidth between pair of nodes.</DIV>
<DIV><BR></DIV>
<DIV>Lustre provides lnet selftest tool for measuring IB bandwidth between pair or group of nodes. Check lustre manual for how tos.</DIV>
<DIV><BR></DIV>
<DIV>Finally, can you post IOR parameters you are using?</DIV>
<DIV><BR></DIV>
<DIV>Cheers,</DIV>
<DIV>Atul</DIV>
<DIV><BR></DIV>
<DIV><BR><BR>Sent from my iPhone</DIV>
<DIV><BR>On Dec 31, 2009, at 5:13 PM, lakshmana swamy <<A href="mailto:klakshman03@hotmail.com">klakshman03@hotmail.com</A>> wrote:<BR><BR></DIV>
<DIV></DIV>
<BLOCKQUOTE>
<DIV> <BR> Dear All,<BR> <BR> Whats the maximum write speed can be achieved with Infiniband QDR HCA card.<BR> <BR> <BR> The Question arised from my lustre setup where Iam not able to exceed 500MB/s (write speed) with two OSSs and one MDS<BR> <BR> Each OSS has four OSTs and one HCA card. I would like to know which is the bottleneck here.<BR> <BR> <BR> I have been using IOR for benchmarking<BR> <BR>Thank you<BR> <BR>LakshmaN<BR><BR>
<HR>
New Windows 7: Simplify what you do everyday. <A href="http://windows.microsoft.com/shop">Find the right PC for you.</A> </DIV></BLOCKQUOTE>
<BLOCKQUOTE>
<DIV><SPAN>_______________________________________________</SPAN><BR><SPAN>Lustre-discuss mailing list</SPAN><BR><SPAN><A href="mailto:Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</A></SPAN><BR><SPAN><A href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</A></SPAN><BR></DIV></BLOCKQUOTE> <br /><hr />New Windows 7: Find the right PC for you. <a href='http://windows.microsoft.com/shop' target='_new'>Learn more.</a></body>
</html>