[Lustre-discuss] IOR performance - Need help

[Fan Yong]

  On 9/14/10 5:57 PM, satish patil wrote:
> Hello,
>
> Recently we installed 6 OSS pairs with 8 OST per pair. Total 48 OST's. Each OST is with 3.7 TB. In all it is 177 TB file system. Lustre version installed is 1.8.1.1 and currently using client based on RHEL 5U2 which is 1.6.x. When running the individual OST test from performance perspecitve we are able to get around 17.5 GB performance. Out target is to cross 10 GBPS write performance using single file w/o -F option avoiding client side cache.  I have reached max to 7.5GB for write performance , but not going beyond. I tried using stripe count as 48 for a single file along with default stripe size which is 1MB. But not able to cross 10 GBPS.
Can you give a detailed description for your system topology? We have 
met customer with more large theory bandwidth, but worse performance, 
because of the unexpected back-end storage performance in parallel.

For I/O performance testing, full stripe maybe not the best choice. 
Using single stripe files, and spreading these relative small files to 
all OSTs evenly, maybe give better result.
> Command line used for running the IOR as follow
> /opt/intel/mpi/bin64/mpirun --totalnum=96 --file=$PBS_NODEFILE --rsh=/usr/bin/ssh -1 --ordered --verbose -l -machinefile $PBS_NODEFILE -np 96 /newScratch/IOR/src/C/IOR.mpiio -a MPIIO -b 22G -C -i 3 -k -t 1m -w -r -R -W -x -N 96 -o /newScratch/hp.stripeC48/IOR.dat
>
> We have used lustre_config to create the file system.
On the other hand, lustre provides basic I/O performance utils (under 
lustre-iokit). You can use them step by step for the basic elements 
performance (like back-end storage, obdfilter, and network), which can 
help you to locate where the performance issues are.


Cheers,
Nasf
> Appriciate your help.
>
> Regards
> SP
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss